Augmented reality hand gesture recognition systems

ABSTRACT

A method, system, apparatus, and/or device for detecting pinch gestures in an augmented reality environment. The method, system, apparatus, and/or device may include: a wearable display, a sensor, and a processing device. The wearable display may be configured to attach to a head of a user and display an augmented reality environment to the user. The sensor may be configured to detect a position of a first digit of a hand of the user and detect a position of a second digit of the hand of the user. The processing device may be configured to: identify a first fingertip of the first digit; identify a second fingertip of the second digit; determine that the first fingertip and the second fingertip are in an open pinch position at a first point in time; and display a cursor at a midpoint between the first fingertip and the second fingertip.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/678,211, filed on May 30, 2018, which are hereby incorporated byreference for all purposes.

BACKGROUND

An increasing number of workers in a variety of industries are assistedby hand-held or wearable computer systems. Rather than using computerkiosks or workstations at locations throughout the work environment, thehand-held or wearable computers allow the worker to move freely aboutthe workspace and retrieve information from computer networks accessibleat their fingertips, while they maintain in proximity to the object usedfor the task.

BRIEF DESCRIPTION OF THE DRAWINGS

The present description will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the present embodiment, which is not to be taken to limitthe present embodiment to the specific embodiments but are forexplanation and understanding.

FIG. 1A illustrates a perspective view of a wearable device being wornby a user, according to an embodiment.

FIG. 1B illustrates a perspective view of the wearable device in FIG.1A, according to an embodiment.

FIG. 2 illustrates an augmented reality system with a user wearing thewearable device 100, according to an embodiment.

FIG. 3 illustrates a system of the wearable device in FIGS. 1A-2,according to an embodiment.

FIG. 4A illustrates an open pinch gesture by a first digit and a seconddigit detected using a touch-free sensor of a wearable device, accordingto an embodiment.

FIG. 4B illustrates a closed pinch gesture by the first digit and thesecond digit detected by the touch-free sensor of a wearable device,according to an embodiment.

FIG. 4C illustrates a cursor location gesture by the first digit and thesecond digit detected using a touch-free sensor of a wearable device,according to an embodiment.

FIG. 5A illustrates a flowchart of a method for detecting a pinch andselect gesture, according to an embodiment.

FIG. 5B illustrates a flowchart for a method of identifying asingle-handed gesture, according to an embodiment.

FIG. 5C illustrates a flowchart for a method of combining gesturesdetected by a wearable device, according to an embodiment

FIG. 6 is a block diagram of a user device with a processing device inwhich implementations of methods, systems, devices, or apparatuses inFIGS. 1-5C may be implemented.

DETAILED DESCRIPTION

The disclosed augmented reality hand gesture recognition systems willbecome better understood through a review of the following detaileddescription in conjunction with the figures. The detailed descriptionand figures provide merely examples of the various embodiments describedherein. Those skilled in the art will understand that the disclosedexamples may be varied, modified, and altered and not depart from thescope of the embodiments described herein. Many variations arecontemplated for different applications and design considerations;however, for the sake of brevity, the contemplated variations may not beindividually described in the following detailed description.

Throughout the following detailed description, examples of variousaugmented reality hand gesture recognition systems are provided. Relatedfeatures in the examples may be identical, similar, or dissimilar indifferent examples. For the sake of brevity, related features will notbe redundantly explained in multiple examples. Instead, the use ofrelated feature names will cue the reader that the feature with arelated feature name may be similar to the related feature in an exampleexplained previously. Features specific to a given example will bedescribed in that particular example. The reader is to understand that agiven feature need not be the same or similar to the specific portrayalof a related feature in any given figure or example.

Individuals that work in industries where their hands must remain freeto perform tasks may use wearable computer systems to retrieve and/orview the information needed to perform the tasks. For example,individuals working in manufacturing facilities may be required tomaintain a clean and hygienic manufacturing area to preventcross-contamination. Additionally, in the manufacturing environment andother environments, it may be required to record that all the steps andinstructions in a defined manufacture procedure were performed. Wearablecomputer systems may aid in recording that all the steps andinstructions were performed and investigate or document deviations takenduring the manufacturing procedure. The records of the manufacturing anddistribution of items with a complete manufacturing history may enable amanufacturer to retain the manufacturing information in a comprehensibleand accessible format in case an investigation is needed to determine aproduct defect or failure. In another example, in retail operationenvironments where sales assistants or inventory control clerks may usewearable computer systems to identify products and/or retrieve productinformation associated with the product.

It may be difficult for individuals to interact with conventionalwearable computer systems. For example, in the clean and hygienicmanufacturing conditions, an individual may not be able to use theirhands to interact with the wearable computer systems in order tomaintain a clean and hygienic environment. Similarly, in a medicalenvironment, medical personnel may not be able to use their hands tointeract with the wearable computer systems in order to avoidintroducing contagions into an operating environment. The conventionalwearable computer systems may use touch-screens or mobile devices forthe users to interact with the system.

Additionally, the conventional systems may use integrated displays orremote displays to display information to the individuals. However, theconventional integrated displays or remote displays may not be viewableto the user. For example, medical personnel may not be able to view theconventional integrated displays or remote displays as they performmedical procedures. Similarly, an individual in a clean and hygienicenvironment performing a manufacturing process may not be able to viewthe conventional integrated displays or remote displays as they performmanufacturing procedures.

Implementations of the disclosure address the above-mentioneddeficiencies and other deficiencies by providing methods, systems,devices, or apparatus to display an augmented reality environment tousers and provide interactive user interfaces. In one embodiment, theaugmented reality systems may be head-mounted systems. The augmentedreality systems may include sensors and/or detector units to detect,gather, and/or transmit data to a processing device. The processingdevice may be an integrated or onboard processing device or an externalor remote processing device. The augmented reality systems may includean augmented reality display to overlay or display information to auser.

In one embodiment, the augmented reality systems may provide a useraccess to remote help, capture images, take sensor measurements, senddata to send to a remote individual, and so forth. For example, anaugmented reality system may transmit a still image or a video from aperspective of the user, also referred to as a “see what I see”perspective. In another example, the augmented reality systems mayoverlay images or information into an augmented reality environmentdisplay for a viewer to see a projected perspective of the images orinformation. An advantage of the augmented reality system may be toprovide a user interface for a user to interact within a variety ofenvironments, including hands-free environments, clean and hygienicenvironments, collaborative environments, remote environments, and soforth.

FIG. 1A illustrates a perspective view of a wearable device 100 beingworn by a user 102, according to an embodiment. The wearable device 100is designed to display an image near an eye or eyes of a user 102 andexecute computer-executable instructions in response user gestures byincorporating motion and image capture features. The wearable device 100may include lens 104 substantially near the eye(s) of the user 102. Thelens 104 may include a transparent or partially transparent screen thatis at least partially disposed within a field of view of the user 102.The screen may display an augmented reality where a user may be able toview augmented reality objects. The screen may have a transparency levelto enable the user 102 to see gestures 106 and real-world objects 108.

FIG. 1B illustrates a perspective view of the wearable device 100 inFIG. 1A, according to an embodiment. Some of the features in FIG. 1B arethe same or similar to some of the features in FIG. 1A as noted by samereference numbers, unless expressly described otherwise.

The wearable device 100 may include a first temple 110, a second temple112, and a bar 114 extending between the two temples 110 and 112. Thefirst temple 110 and second temple 112 extend from the wearable device100 to rest on the ears of the user 102 in FIG. 1A to provide supportfor the wearable device 100.

The wearable device 100 may include a processing device 116, the lens104, a first image generator 118, a second image generator 120, a motioncapture device 124, and a microphone 122. The wearable device 100 may beconfigured to capture user gestures, along with other objects within acaptured scene, and execute computer processes or commands in responseto the captured objects. This may involve manipulating a displayproduced by a first image generator 118 and/or a second image generator120 on lens 104. For example, the first image generator 118 may displaya first virtual object or portion of a virtual object on a first portionof the lens 104. In another example, the second image generator 120 maydisplay a second virtual object or portion of a virtual object on asecond portion of the lens 104. In another example, the lens 104 may bea single continuous lens extending across the bar 114. In anotherexample, the lens 104 may include multiple lens or portions of lensextending across the bar 114. The first virtual object and/or the secondvirtual object includes text, a graphical object, a video, a live datasteam, and so forth.

The wearable device 100 may additionally execute computer processes inresponse to audio captured by the microphone 122. By incorporating thesecapturing technologies, the wearable device 100 may display and controlcomputer images and processes. The processing device 116 may besupported by the bar 114. The processing device 116 may be configured toexecute computer-executable instructions and control the electronicelements of the wearable device 100.

The processing device 116 may acquire data from other elements of thewearable device 100 or from external sources and execute computerexecutable code in response to this data. For example, the processingdevice 116 may be configured to acquire data from the motion capturedevice 124 such as data that corresponds to a user gesture. Theprocessing device 116 may additionally or alternatively acquire datafrom the microphone 122. In some examples, the processing device 116 mayacquire data from a separate device, such as a portable music player, apersonal data assistant (PDA), a smartphone, a global positioning system(GPS), or the like.

The processing device 116 may be in coupled to the first image generator118 and/or the second image generator 120 and may instruct the firstimage generator 118 and/or the second image generator 120 to generateand manipulate a display projected on lens 104. In one example, theprocessing device 116 may generate a user interface on the lens 104. Theprocessing device 116 may acquire data from other elements or sensors ofthe wearable device 100 and manipulate the user interface in response tothis data.

The first image generator 118 and/or the second image generator 120 maybe attached to the first temple 110 and may be substantially alignedwith the lens 104. The wearable device 100 may include additional imagegenerators to augment the expanse of generated images over the lens 104surface area. The first image generator 118 and/or the second imagegenerator 120 may be configured to display images on the lens 104 facingan eye or eyes of the user 102 in FIG. 1A in response to instructionsexecuted by the processing device 116. These images often may overlayand/or augment a naturally viewed scene within a field of vision of theuser 102. The first image generator 118 and/or the second imagegenerator 120 may display opaque and/or partially transparent images onthe lens 104.

The motion capture device 124 may be connected to the bar 114. In oneembodiment, the motion capture device 124 may be oriented away from theuser 102 when the user 102 is wearing the wearable device 100 to captureimages and motion occurring beyond the lens 104. In another embodiment,the motion capture device 124 may include an image sensor 126 with acamera, a light source 128, and a depth sensor 130. The motion capturedevice 124 may capture images that include at least a portion of theenvironment surrounding the wearable device 100 (such as a head-mounteddevice or a head-mounted display).

The image sensor 126 may be connected to the bar 114. In one embodiment,the image sensor 126 may be oriented away from the user 102 as the user102 wears the wearable device 100. The image sensor 126 may beconfigured to capture an image from a scene and communicate datacorresponding to the captured image to the processing device 116. Theimage sensor 126 may capture light within the visual spectrum and/orlight outside the visual spectrum. For example, the image sensor 126 maycapture infrared or ultraviolet light.

The light source 128 may be connected to the bar 114. The light source128 may be substantially aligned with the image sensor 126. The lightsource 128 may be configured to project light in a defined direction.This light is reflected from an object and may be captured by the motioncapture device 124. The reflected light may allow the motion capturedevice 124 to more accurately capture images and motion of objects orthe surrounding environment. The light source 128 may project visuallight and/or light outside the visual spectrum.

The depth sensor 130 may be connected to the bar 114. The depth sensor130 may be configured to capture images and motion from a scene at adifferent angle than image sensor 126. The data captured from thissecond viewpoint may allow the processing device 116 to compare the datareceived from the depth sensor 130 to the data received from the imagesensor 126 to better detect a depth of objects in the environmentsurrounding the wearable device 100.

FIG. 2 illustrates an augmented reality system 200 with a user 102wearing the wearable device 100, according to an embodiment. Some of thefeatures in FIG. 2 are the same or similar to some of the features inFIGS. 1A and 1B as noted by same reference numbers, unless expresslydescribed otherwise. The user may be located along a first plane 204.The wearable device 100 may include a lens 104 to display a virtualobject to an eye or eyes of the user 102. For example, the wearabledevice 100 may include a first portion of the lens 104 that may displaya first image or virtual object to a first eye of the user 102 and asecond portion of the lens 104 that may display a second image orvirtual object to a second eye of the user 102. When combined oraggregated, the first image or virtual object and the second image orvirtual object may form an augmented reality environment to project oneor more virtual or augmented reality objects to the user 102 at definedlocations within a physical environment surrounding the user 102.

The wearable device 100 may display multiple images to project one ormore virtual or augmented reality objects 208 within a physicalenvironment surrounding the user 102. In one embodiment, the user andthe wearable device 100 may be located along the first plane 204 and theaugmented reality object 208 may be displayed to appear at a defineddistance from the user 102 at a second plane 206. In one embodiment, thewearable device 100 may include a position sensor, such as a GPS deviceor a position transceiver. For example, the wearable device 100 may be apair of glasses, smart glasses, a face shield, and so forth that may beworn by the user 102 and include multiple sensors, including theposition sensor. In another embodiment, the position sensor may separatefrom the wearable device 100 and be attached to the body of the user102.

In one example, the position sensor may be positioned proximate to aline of sight of the user 102. In another embodiment, an alignmentsensor may measure a difference between a position of the user 102indicated by the position sensor and the actual location of the user'seyes and generates a view aligned to the user's line of sight. In oneembodiment, the lens 104 in FIGS. 1A and 1B may project a virtual oraugmented reality environment or a virtual or augmented reality object208 at a set focal distance from the user 102. In one example, the focaldistance may be along the second plane 206. The virtual or augmentedreality environment or the virtual or augmented reality object 208 maybe a user interface that enhances or supplements the user's interactionsin the physical environment and/or physical objects approximate orsurrounding the user.

FIG. 3 illustrates a system 300 of the wearable device 100 in FIGS.1A-2, according to an embodiment. The system 300 may include a detector302, sensors 304, a transceiver 306, an augmented reality display 308, alocal processing device 310, a remote processing device 312, and anoptimizer 314. In one embodiment, the detector 302 and/or the sensors304 may take measurements and collect data from a physical environmentapproximate to the wearable device 100. In another embodiment, the localprocessing device 310 may be a local processing device attached to orintegrated into the wearable device 100.

The local processing device 310 may be positioned proximate to the userthat interacts with a remote processing device 312. In anotherembodiment, the remote processing device 312 may be a remote processingdevice that may be external to the wearable device 100. The transceiver306 may receive instructions from the local processing device 310 toproject the augmented display 20. The transceiver 306 may measurecharacteristics from the physical environment and communicate the datato the local processing device 310 and an optimizer 314.

The optimizer 314 may be in communication with the remote processingdevice 312, the local processing device 310, and the transceiver 306.The transceiver 306 may measure or detect user characteristics orgestures including head position, movement, speed, etc. In one example,using gestures, the user may open applications to be displayed in theaugmented environment. These applications may include task flows,attached documents, and so forth. The gestures may allow the user toaccess the applications and execute instructions such as selectinstructions, open instructions, scrolling instructions, movementinstructions, and so forth. Alternatively, the gestures may be used tooperate on the icon representing the application by changing the size(zooming in and out), color, sound, and so forth. The gestures mayinclude the user performing scrolling gestures, tapping gestures, orclicking gestures with respect to a virtual object. The scrollinggestures may include body motions, head motions, hand motions, and soforth.

FIG. 4A illustrates an open pinch gesture 402 by a first digit 404 and asecond digit 406 detected using a touch-free sensor of a wearabledevice, according to an embodiment. In one embodiment, the open pinchgesture 402 may be associated with a first input 410 by a third digit412 that is detected by a touch sensor 412. FIG. 4B illustrates a closedpinch gesture 414 by the first digit 404 and the second digit 406detected by the touch-free sensor of the wearable device, according toan embodiment. In another embodiment, the closed pinch gesture 414 maybe associated with a second input 416 by a third digit 410 that isdetected by the touch sensor 412. The first digit 404, the second digit406, and/or the third digit 410 may be fingers of one or more hands,styluses, and so forth. In one example, the first digit 404 may be afirst finger of a hand and the second digit 406 may be a second fingerof the hand. In another example, the third digit 410 may be the firstdigit 404, the second digit 406, or another digit of the hand.

In one embodiment, when the processing device detects the open pinchgesture 402 followed by the open pinch gesture 402 changing to theclosed pinch gesture 414, a processing device of the wearable device mayexecute a tap instruction (such as a single tap or a double tap)associated with the user tapping on the touch sensor 412. For example,the open pinch gesture 402 may be associated with a hovering gesture ofthe third digit 410 hovering over the touch sensor 412, as shown in FIG.4A. The open pinch gesture 402 changing to the closed pinch gesture 414,as shown in FIG. 4B, may be associated with the third digit 410 movingfrom the hovering position to tapping on the touch sensor 412.

In another embodiment, when the processing device detects (using asensor) the closed pinch gesture 414 followed by the closed pinchgesture 414 changing to the open pinch gesture 402, the processingdevice may execute a swipe instruction associated with the user swipingon the touch sensor 412. In another embodiment, when the processingdevice detects that the closed pinch gesture 414 is performed for athreshold period of time (where the fingers remain pinched), theprocessing device may execute a press and hold instruction. In oneexample, the threshold period of time for the fingers to remain pinchedafter detecting closed pinch gesture 414 may be 800 milliseconds. In oneexample, the threshold period of time for the fingers to remain pinchedafter detecting closed pinch gesture 414 may be at least 0.5 seconds.

In one embodiment, the wearable device may include a camera configuredto capture the open pinch gesture 402 and/or the closed pinch gesture414. For example, the camera may be configured to detect the positionsof the first digit 404 and/or the second digit 406 by generating a handskeletal structure information and applying one or more geometricfilters on top of hand skeletal structure information. For example, theprocessing device may execute a hand skeleton detection algorithm (HSDA)that identifies hand skeleton key point locations in an image(s) orvideo captured by the camera and determines geometric information of theimage(s) or frame(s) of the video using a depth sensor or depthmeasurements.

The hand skeleton key point may include fingertips, finger bases, a palmcenter, wrist points, an arm-point, and so forth. When the processingdevice identifies the hand skeleton structure information, theprocessing device may use the hand skeleton key points to identify thepositions of the first digit 404, the second digit 406, and/or the thirddigit 410 to determine the open pinch gesture 402 and/or the closedpinch gesture 414. For example, the processing device may analyze animage or a frame to determine the positions of the first digit 404 andthe second digit 406 relative to the palm center and/or the wrist point.

In another embodiment, the processing device may detect multiple handsperforming the open pinch gesture 402 and/or the closed pinch gesture414. The processing device may determine the number of hands within athreshold range of a sensor (such as a camera) and monitor the gesturesmade by each hand within the threshold range. For example, when theprocessing device detects two hands within the threshold range, theprocessing device may monitor both hands for gestures and when either orboth hands perform gestures associated with an instruction, theprocessing device may execute the associated instruction. In oneembodiment, the threshold range may be between 12 inches from the sensorand 48 inches from the sensor. In another embodiment, the thresholdrange may be 24 inches from the sensor to 84 inches from the sensor. Inanother embodiment, the processing device may define a center point at adefined distance from the sensor and the threshold range may be athreshold distance from the center point. For example, the center pointmay be 36 inches from the sensor and the threshold distance may be 0inches to 36 inches from the center point.

In another embodiment, when the processing device detects 2 hands withina threshold distance from a two-handed rotation axis point (THRA point)where both hands are the same or similar distances from the THRA point,the processing device may execute a rotate instruction for a virtualobject. In one embodiment, the processing device may determine whetherthe two hands are within the threshold distance or a distance of thehand (DoH) of the THRA point using a depth sensor. For example, when ahand of the user is moving along an X axis, a Y axis, or a Z axis, thedepth sensor may detect a range of the hand along the X axis, the Yaxis, or the Z axis relative to the THRA point. In one example, therange may be from 10 centimeters (cm) to 100 cm. In one example, whenthere are N objects within a first threshold range along the X axis andY axis but at different ranges along the Z axis, the processing devicemay divide or map the N objects into different quadrants based on wherethe object lies along the Z axis. The processing device may determinewhere the two hands are along the Z axis to determine the quadrant thetwo hands are within and then rotate the object within that quadrant.

In one embodiment, the similar distance of both hands from the THRApoint may be that both hands are within 6 inches of the same distancefrom the center point. In another embodiment, the similar distance ofboth hands from the THRA point may be when both hands are equidistantfrom a defined point or location in space, such as a midpoint along aline. The processing device may define the THRA point as the axis aroundwhich a rotation may be performed. The THRA point may be a midpoint ofthe two pinch locations.

In one example, when the processing device detects open pinch gestures402 and/or closed pinch gestures 414 by the digits of both hands movingin same cyclic directions (e.g. clockwise or counterclockwise) aroundthe THRA point, the processing device may execute the rotate instructionto rotate the virtual object. In another embodiment, when both handsremain within the same or similar distance from the THRA point and areeach performing the open pinch gestures 402 and/or closed pinch gestures414, the processing device may define the THRA point as being stable atthe defined location and may execute the rotate instruction. When theprocessing device determines that the THRA point is not stable, theprocessing device may not execute the rotate instruction.

In another embodiment, when the processing device detects the two handsperforming the closed pinch gesture 414 and moving in contracting orexpanding directions relative to the THRA point, the processing devicemay execute a zoom in or zoom out instruction, respectively. In anotherembodiment, when the processing device detects the closed pinch gesture414 in conjunction with rotating the hands about one of an X axis, a Yaxis, or a Z axis, the processing device may execute a rotatinginstruction for a virtual object in an augmented reality environment. Inanother embodiment, when the processing device detects the closed pinchgesture 414 in conjunction with a hand moving along a Z axis, theprocessing device may execute a zoom in instruction or a zoom outinstruction based on the direction the hand moves along the Z axis. Inone example, the Z-axis may be perpendicular to a display of thewearable device. In another embodiment, a hand distance between twohands of a user with respect to the display of the wearable device maybe used to identify which object within an augmented reality environmentthe user desires to interact when there are multiple objects in closeX/Y position but different Z position.

In one embodiment, to identify different pinch gestures for differentindividuals, as different individuals may pinch differently, theprocessing device may use a midpoint a fingertip of the first digit 404and a fingertip of the second digit 406 when identifying the open pinchgesture 402 and the closed pinch gesture 414. The processing device maymeasure the movement of the fingertips relative to the midpoint todetermine if the first digit 404 and the second digit 406 are performingthe open pinch gesture 402, the closed pinch gesture 414, or anothergesture. For example, an individual may move a fingertip, such as athumb fingertip, while making a pinching gesture and such movement maycause the gesture movement to be unstable. In this example, theprocessing device may use the midpoint between the fingertips to avoidinstability in measuring the open pinch gesture 402 or the closed pinchgesture 414. In another example, when a user turns their hand to anangle that two fingers are facing forwards (away from the user), asensor (such as a camera or a depth sensor) may not be able to detect aposition where the fingertips touch each other to perform the open pinchgesture 402 and/or the closed pinch gesture 414. In one embodiment, todetect the open pinch gesture 402 and/or the closed pinch gesture 414when the fingers of the user are facing forward before the two fingerstouch each other, the processing device may determine a pinch intent ofthe user. To determine the pinch intent, the depth sensor may identifyanother digit of the individual (such as an index finger) and compare aposition of the other digit to a database indicating positions of theother digit. The processing device may then estimate whether the otherdigit indicates that the user is performing the open pinch gesture 402and/or the closed pinch gesture 414 based on the position of the otherdigit.

In another example, when the first digit 404 and the second digit 406fingers are facing away from a face of the user, an image from thecamera of the wearable device may display the first digit 404 and thesecond digit 406 as viewable as pointing a vertical line or a ‘1’symbol. In the image(s) and/or frame(s) where the side profile of thefirst digit 404, the second digit 406, and/or the other digits of a handmay not be viewable based on the HSDA, the processing device mayestimate the position of the first digit 404, the second digit 406,and/or the other digits of a hand. For example, when an individualperforms a pinch, the images or frames may illustrate a sequence of thefirst digit 404 with at least a portion of the fingertip graduallymoving downward relative to the other digits of the hand until the firstdigit 404 stops moving downward.

In one embodiment, the first digit 404 may stop moving downward when thefirst digit 404 is a defined distance from the second digit 406. Theprocessing device may establish a midpoint between the first digit 404and the second digit 406 based on a relative distance between the seconddigit 406 and one or more of the other digits of the hand. In oneembodiment, the first digit 404 may stop moving downward when the firstdigit 404 is in contact with the second digit 406. The processing devicemay determine the contact between the first digit 404 and the seconddigit 406 based on a relative distance between the second digit 406 andone or more of the other digits of the hand. The contact between thefirst digit 404 and the second digit 406 may correspond with a tappinginstruction as discussed above.

In another embodiment, the processing device may determine the openpinch gesture 402 changing to the closed pinch gesture 414 (alsoreferred to as a pinch intent) by comparing positions of the first digit404 and the second digit 406 to a training dataset of the averagepositions of the first digit 404 and the second digit 406 given a palmsize of the user. For example, based on training dataset, the processingdevice may determine an average distance range between the first digit404 and the second digit 406 when the first digit 404 and the seconddigit 406 are in the open pinch position 402 corresponding to a hoveringposition above the touch sensor 412. The processing device may alsodefine a buffer value to reduce or eliminate false positives. When thedistance between the first digit 404 and the second digit 406 becomesless than the average distance range minus the buffer, the processingdevice may determine that a pinch intent has occurred.

FIG. 4C illustrates a cursor location gesture 418 by the first digit 404and the second digit 406 detected using a touch-free sensor of awearable device, according to an embodiment. Some of the features inFIG. 4C are the same or similar to some of the features in FIGS. 4A and4B as noted by same reference numbers, unless expressly describedotherwise.

As discussed above, the processing device may use the HSDA to detect keypoints of an individual's hand(s). In one embodiment, the processingdevice may use the key points to determine a location to display acursor 420 in an augmented reality environment. In one example, theprocessing device may determine a location of the cursor 420 relative tothe fingertip of the first digit 404, the fingertip of the second digit406, and a palm-center (defined as a midpoint) to remove variability orinstability in fingertip motions.

In one embodiment, the processing device may determine a midpointbetween a fingertip of the first digit 404 and a fingertip of the seconddigit 406 when the fingertip of the first digit 404 and the fingertip ofthe second digit 406 are in an open position. For example, when thefingertip of the first digit 404 and the fingertip of the second digit406 form an open pinch gesture 402 where the first digit 404 and thesecond digit 406 form an open C shape, the processing device maydetermine a midpoint between the fingertip of the first digit 404 andthe fingertip of the second digit 406 to execute a cursor locationinstruction. The cursor location instruction may position a cursor 420in an augmented reality environment at the midpoint between thefingertip of the first digit 404 and the fingertip of the second digit406. In one embodiment, the cursor location instruction may be executedto define the location of the cursor 420 so that when a tappinginstruction is performed by the user as discussed in FIGS. 4A and 4B,the processing device may define what virtual object the user is tappingon in the augmented reality environment. In another embodiment, themidpoint may be defined at a beginning of the transition from the openpinch gesture 402 in FIG. 4A to the closed pinch gesture 414 in FIG. 4Bsuch that the cursor 420 remains steady relative to a wrist point of ahand throughout the transition.

In another embodiment, to identify different pinch gestures fordifferent individuals, when the fingers of the user are within a defineddistance of the wearable device to identify pinch intention, the cursor420 in the augmented reality environment may be positioned between thetwo fingers of the user performing the open pinch gesture 402 and/or theclosed pinch gesture 414. For example, when the individual starts theopen pinch gesture 402 or the closed pinch gesture 408, the cursor 420may stay between the fingers as the first digit 404 and/or the seconddigit 406 move towards each other and when the first digit 404 istouching the second digit 406, the processing device may render a steadycursor 420 that the user can aim and select objects with.

As the fingers start pinching in, the cursor 420 may remain visiblystable regardless of how the two fingers are moving, as long as thewrist or hand is still or relatively still. For example, the user maysee a stable cursor 420 that does not shift during the pinch gestureprocess and they can aim and select objects within the augmented realityenvironment using the cursor 420. In one example, if the wrist or thehand of the user moves during the pinch gesture process, the processingdevice may move the cursor 420 according to the movement and/or rotationof the wrist or the hand. The midpoint may then be correlated to acursor 420 such that as the midpoint moves in the physical environmentthe cursor 420 may move in the augmented reality environment.

FIG. 5A illustrates a flowchart 500 of a method for detecting a pinchand select gesture, according to an embodiment. In one example, a usermay form a C shape with a first digit and a second digit of their righthand or their left hand and hovers over a user interface (UI) elementthe user intends to select. When the user pinches the fingertips of thefirst digit and the second digit together to transition from the C shapeto an O shape of the first digit and the second digit, a processingdevice may execute a UI element selection instruction.

The method may include identifying at least two distal ends of a handfor a single-handed gesture (block 502). In one example, the two distalends may be the fingertips of the first digit and the second digit. Thefirst digit or the second digit may be a finger or thumb of the user. Asdiscussed above, the distal ends or fingertips may be identified basedon skeletal key points in a hand skeleton in different images ordifferent frames of a video. The processing device may identify handskeleton keypoint locations using the images or frames and determine ageometry of the distal ends using a depth sensor and or the HSDA.

The method may include determining that the two distal ends are within athreshold distance of each other (block 504). In one embodiment, thethreshold distance may be based on a Euclidean distance between the twodistal ends in a three dimensional (3D) space. In one embodiment, thethreshold distance may be a distance of 1 centimeter (cm). In oneexample, when the two distal ends are within 1 cm of each other, the twodistal points may be within the threshold distance. The thresholddistance of 1 cm is not intended to be limiting and the thresholddistance may be other distances, such as 0.5 cm or 2 cm.

The method may include determining whether a duration that the two endsare within the threshold distance for a threshold amount of time (block506). In one example, the threshold amount of time may be 500milliseconds. In another example, the threshold amount of time may be0.5 seconds. The threshold amount of time is not intended to be limitingand the threshold amount of time may be other amounts of time, such as 1second or 250 milliseconds.

In another example, the threshold amount of time may vary for differentinstructions associated with the two distal ends being within thethreshold distance of each other. For example, the threshold may be lessthan or equal to 500 milliseconds where when the user holds the twodistal ends together for less than or equal to 500 milliseconds theprocessing device may execute a select instruction to select an objector element in the augmented reality environment. In another example, thethreshold may be greater than 500 milliseconds where when the user holdsthe two distal ends together for an amount of time greater than 500milliseconds the processing device may execute a hold-and-draginstruction to select the object or element and move the object orelement from a first location to a second location in the augmentedreality environment.

The method may include selecting a user interface (UI) element (block508). In one embodiment, the processing device may perform the method inflowchart 500 for both hands of a user where one distal end of each handor a midpoint between both hands is used to execute an instruction. In adual hand embodiment, a distal end of each hand is identified and thetwo-handed instruction may be performed when both distal ends perform agesture. A single hand instruction using two distal ends of one hand mayinclude a finger tap instruction, scroll instruction, a long pressinstruction, a repositioning instruction, a selection instruction, andso forth. The selection instruction or finger tap instruction may be aninstruction where the user pinches and selects an item in the augmentedreality environment or physical environment and an application, data,and so forth associated with the item is displayed in the augmentedenvironment.

The repositioning instruction may include the processing device using acamera identifying a first location where the two distal ends camewithin the threshold distance (also referred to as a pinch gesture) anddetermine the two distal ends then move to a second location beforemoving outside the threshold distance from each other. The UI elementmay be repositioned in the augmented reality environment from the firstlocation to the second location. A two hand instruction using two distalends of two hand may include zoom instruction, rotate instruction, andso forth.

In one example, to the select the UI element, one of the distal ends mayhover over a UI element to be selected, the other distal end may tap theselecting distal end, and when the duration that the distal ends arewithin a threshold distance of each other exceeds a threshold amount oftime the UI element may be selected.

FIG. 5B illustrates a flowchart 520 for a method of identifying asingle-handed gesture, according to an embodiment. The method mayinclude detecting, by a sensor, a position of a first digit of a hand ofthe user (block 522). The method may include detecting, by the sensor, aposition of a second digit of the hand of the user (block 524). Themethod may include identifying, by a processing device, a firstfingertip of the first digit (block 526). The method may includeidentifying a second fingertip of the second digit (block 528). Themethod may include determining that the first fingertip and the secondfingertip are in an open pinch position at a first point in time (block530). The method may include determining that the first fingertip andthe second fingertip are in a closed pinch position at a second point intime, wherein the second point in time is after the first point in time(block 532). The method may include determining that the first fingertipand the second fingertip are in the closed pinch position for athreshold period of time (block 534). The method may include in responseto that the first fingertip and the second fingertip are in the closedpinch position for a threshold period of time, executing a tapinstruction associated with the closed pinch position (block 536). Inone example, the tap instruction may correspond with a tap instructionexecuted on a touch sensor of another device.

FIG. 5C illustrates a flowchart 540 for a method of combining gesturesdetected by a wearable device, according to an embodiment. The methodmay include identifying two distal ends of a first hand and two distalends of a second hand (block 542). In one example, a first cursor may bedisplayed between the two distal ends of the first hand and a secondcursor may be displayed between the two distal ends of the second hand.The method may include detecting the two distal ends of the first handare in a closed pinch position and the two distal ends of the secondhand are in the closed pinch position (block 544). In one example, whenthe two distal ends of the first hand are in the closed pinch positionthe first cursor may become a first engagement cursor to indicate afirst point a UI element is engaged or selected and when the two distalends of the second hand are in the closed pinch position the secondcursor may become a second engagement cursor to indicate a second pointa UI element is engaged or selected.

The method may include determining a position of the first and secondhand over a period of time and executing an instruction based on theposition of first and second hands over the period of time while thedistal ends of the first and second hands are in the closed pinchposition (block 546). In one example, the instruction may be a zoominstruction to zoom in or zoom out on the UI element. For example, asthe first and second hands move closer to each other the UI element maybe enlarged to zoom in and as the first and second hands move away fromeach other the UI element may be shrunk to zoom out, or vise versa. Inanother example, the amount that the UI element is enlarged or shrunkenmay be relative to a distance the two distal ends of the first hand andthe second hand are to a midpoint or a central point between the twodistal ends of the first hand and the second hand. The relative distancetraveled by the two hands may determine the level of zooming (i.e.enlargement or shrinkage). In another example, as the first hand and thesecond hand remain at approximately the same distance from each otherand are rotated clockwise or counterclockwise, the selected UI elementmay be rotated in the augmented display along an axis correspondingclockwise or counterclockwise movement. In another example, the rotationof the selected UI element may be a three-dimensional rotation whereasthe two distal ends of the first hand and the second hand are rotated ormoved within three-dimensional space the UI element may correspondinglyrotate in the three-dimensional space about an axis.

The method may include determining when the distal ends of the firsthand and/or the distal ends of the second hand disengage from the closedpinch position (block 548). The method may include, in response to thedisengagement, ceasing to execute instructions associated with themovements of both hands (block 550).

FIG. 6 is a block diagram of a user device 600 with a processing devicein which implementations of methods, systems, devices, or apparatuses inFIGS. 1-5C may be implemented. The user device 600 may display and/orimplement the method and GUIs of FIGS. 1A-5C. The user device 600 may beany type of computing device such as an electronic book reader, a PDA, amobile phone, a laptop computer, a portable media player, a tabletcomputer, a camera, a video camera, a netbook, a desktop computer, agaming console, a DVD player, a computing pad, a media center, and thelike. The user device 600 may be any portable or stationary user device.For example, the user device 600 may be an intelligent voice control andspeaker system. Alternatively, the user device 600 can be any otherdevice used in a WLAN network (e.g., Wi-Fi® network), a WAN network, orthe like.

The user device 600 includes one or more processing device(s) 610, suchas one or more CPUs, microcontrollers, field programmable gate arrays,or other types of processing devices. The user device 600 also includessystem memory 602, which may correspond to any combination of volatileand/or non-volatile storage mechanisms. The system memory 602 storesinformation that provides operating system 604, various program modules606, program data 608, and/or other components. In one implementation,the system memory 602 stores instructions of the methods in FIGS. 5A-5Cas described herein. The user device 600 performs functions by using theprocessing device(s) 610 to execute instructions provided by the systemmemory 602.

The user device 600 also includes a data storage device 616 that may becomposed of one or more types of removable storage and/or one or moretypes of non-removable storage. The data storage device 616 includes acomputer-readable storage medium 618 on which is stored one or more setsof instructions embodying any of the methodologies or functionsdescribed herein. Instructions for the program modules 606 may reside,completely or at least partially, within the computer-readable storagemedium 618, system memory 602 and/or within the processing device(s) 610during execution thereof by the user device 600, the system memory 602and the processing device(s) 610 also constituting computer-readablemedia. The user device 600 may also include one or more input devices612 (keyboard, mouse device, specialized selection keys, etc.) and oneor more output devices 614 (displays, printers, audio output mechanisms,etc.).

The user device 600 further includes modem 620 to allow the user device600 to communicate via a wireless network(s) (e.g., such as provided bythe wireless communication system) with other computing devices, such asremote computers, an item providing system, and so forth. The modem 620can be connected to zero or more RF modules 622. The zero or more RFmodules 622 can be connected to zero or more coupler circuitry 624. TheRF modules 622 and/or the coupler circuitry 624 may be a WLAN module, aWAN module, PAN module, or the like. Antenna 626 is coupled to thecoupler circuitry 624, which is coupled to the modem 620 via the RFmodules 622. The modem 620 allows the user device 600 to handle bothvoice and non-voice communications (such as communications for textmessages, multimedia messages, media downloads, web browsing, etc.) witha wireless communication system. The modem 620 may provide networkconnectivity using any type of mobile network technology including, forexample, cellular digital packet data (CDPD), general packet radioservice (GPRS), EDGE, universal mobile telecommunications system (UMTS),1 times radio transmission technology (1×RTT), evaluation data optimized(EVDO), high-speed downlink packet access (HSDPA), Wi-Fi® technology,Long Term Evolution (LTE) and LTE Advanced (sometimes generally referredto as 4G), etc.

The modem 620 may generate signals and send these signals to antenna 626via coupler circuitry 624 as described herein. User device 600 mayadditionally include a WLAN module, a GPS receiver, a PAN transceiverand/or other RF modules. The coupler circuitry 624 may additionally oralternatively be connected to one or more of coupler arrays. The antenna626 may be configured to transmit in different frequency bands and/orusing different wireless communication protocols. The antenna 626 may bedirectional, omnidirectional, or non-directional antennas. In additionto sending data, the antenna 626 may also receive data, which is sent toappropriate RF modules 622 connected to the antenna 626.

In one implementation, the user device 600 establishes a firstconnection using a first wireless communication protocol, and a secondconnection using a different wireless communication protocol. The firstwireless connection and second wireless connection may be activeconcurrently, for example, if a user device is downloading a media itemfrom a server (e.g., via the first connection) and transferring a fileto another user device (e.g., via the second connection) at the sametime. Alternatively, the two connections may be active concurrentlyduring a handoff between wireless connections to maintain an activesession (e.g., for a telephone conversation). Such a handoff may beperformed, for example, between a connection to a WLAN hotspot and aconnection to a wireless carrier system. In one implementation, thefirst wireless connection is associated with a first resonant mode of anantenna structure that operates at a first frequency band and the secondwireless connection is associated with a second resonant mode of theantenna structure that operates at a second frequency band. In anotherimplementation, the first wireless connection is associated with a firstantenna element and the second wireless connection is associated with asecond antenna element. In other implementations, the first wirelessconnection may be associated with a media purchase application (e.g.,for downloading electronic books), while the second wireless connectionmay be associated with a wireless ad hoc network application. Otherapplications that may be associated with one of the wireless connectionsinclude, for example, a game, a telephony application, an Internetbrowsing application, a file transfer application, a global positioningsystem (GPS) application, and so forth.

Though modem 620 is shown to control transmission and reception via theantenna 626, the user device 600 may alternatively include multiplemodems, each of which is configured to transmit/receive data via adifferent antenna and/or wireless transmission protocol.

The user device 600 delivers and/or receives items, upgrades, and/orother information via the network. For example, the user device 600 maydownload or receive items from an item providing system. The itemproviding system receives various requests, instructions and other datafrom the user device 600 via the network. The item providing system mayinclude one or more machines (e.g., one or more server computer systems,routers, gateways, etc.) that have processing and storage capabilitiesto provide the above functionality. Communication between the itemproviding system and the user device 600 may be enabled via anycommunication infrastructure. One example of such an infrastructureincludes a combination of a wide area network (WAN) and wirelessinfrastructure, which allows a user to use the user device 600 topurchase items and consume items without being tethered to the itemproviding system via hardwired links. The wireless infrastructure may beprovided by one or multiple wireless communications systems, such as oneor more wireless communications systems. One of the wirelesscommunication systems may be a wireless local area network (WLAN)hotspot connected to the network. The WLAN hotspots can be created byproducts based on IEEE 802.11x standards for the Wi-Fi® technology byWi-Fi® Alliance. Another of the wireless communication systems may be awireless carrier system that can be implemented using various dataprocessing equipment, communication towers, etc. Alternatively, or inaddition, the wireless carrier system may rely on satellite technologyto exchange information with the user device 600.

The communication infrastructure may also include acommunication-enabling system that serves as an intermediary in passinginformation between the item providing system and the wirelesscommunication system. The communication-enabling system may communicatewith the wireless communication system (e.g., a wireless carrier) via adedicated channel, and may communicate with the item providing systemvia a non-dedicated communication mechanism, e.g., a public Wide AreaNetwork (WAN) such as the Internet.

The user device 600 is variously configured with different functionalityto enable consumption of one or more types of media items. The mediaitems may be any type of format of digital content, including, forexample, electronic texts (e.g., eBooks, electronic magazines, digitalnewspapers, etc.), digital audio (e.g., music, audible books, etc.),digital video (e.g., movies, television, short clips, etc.), images(e.g., art, photographs, etc.), and multi-media content. The userdevices 600 may include any type of content rendering devices such aselectronic book readers, portable digital assistants, mobile phones,laptop computers, portable media players, tablet computers, cameras,video cameras, netbooks, notebooks, desktop computers, gaming consoles,DVD players, media centers, and the like.

In the above description, numerous details are set forth. It will beapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that implementations may be practicedwithout these specific details. In some instances, well-known structuresand devices are shown in block diagram form, rather than in detail, inorder to avoid obscuring the description.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to thedesired result. The steps are those requiring physical manipulations ofphysical quantities. Usually, though not necessarily, these quantitiestake the form of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “inducing,” “parasitically inducing,” “radiating,”“detecting,” determining,” “generating,” “communicating,” “receiving,”“disabling,” or the like, refer to the actions and processes of acomputer system, or similar electronic computing device, thatmanipulates and transforms data represented as physical (e.g.,electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Implementations also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in acomputer-readable storage medium, such as, but not limited to, any typeof disk including floppy disks, optical disks, CD-ROMs andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present implementations are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the present invention as described herein. It should alsobe noted that the terms “when” or the phrase “in response to,” as usedherein, should be understood to indicate that there may be interveningtime, intervening events, or both before the identified operation isperformed.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the present implementations should,therefore, be determined with reference to the appended claims, alongwith the full scope of equivalents to which such claims are entitled.

In the above description, numerous details are set forth. It will beapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that implementations may be practicedwithout these specific details. In some instances, well-known structuresand devices are shown in block diagram form, rather than in detail, inorder to avoid obscuring the description.

Although the implementations may be herein described with reference tospecific integrated circuits, such as in computing platforms ormicroprocessing devices, other implementations are applicable to othertypes of integrated circuits and logic devices. Similar techniques andteachings of implementations described herein may be applied to othertypes of circuits or semiconductor devices. For example, the disclosedimplementations are not limited to desktop computer systems orUltrabooks™ and may be also used in other devices, such as handhelddevices, tablets, other thin notebooks, systems on a chip (SOC) devices,and embedded applications. Some examples of handheld devices includecellular phones, Internet protocol devices, smartphones, digitalcameras, personal digital assistants (PDAs), and handheld PCs. Embeddedapplications typically include a microcontroller, a digital signalprocessing device (DSP), a system on a chip, network computers (NetPC),set-top boxes, network hubs, wide area network (WAN) switches, or anyother system that may perform the functions and operations taught below.

Although the implementations are herein described with reference to aprocessing device or processing device, other implementations areapplicable to other types of integrated circuits and logic devices.Similar techniques and teachings of implementations of the presentinvention may be applied to other types of circuits or semiconductordevices that may benefit from higher pipeline throughput and improvedperformance. The teachings of implementations of the present inventionare applicable to any processing device or machine that performs datamanipulations. However, the present invention is not limited toprocessing devices or machines that perform 512 bit, 256 bit, 128 bit,64 bit, 32 bit, and/or 16 bit data operations and may be applied to anyprocessing device and machine in which manipulation or management ofdata is performed. In addition, the following description providesexamples, and the accompanying drawings show various examples for thepurposes of illustration. However, these examples should not beconstrued in a limiting sense as they are merely intended to provideexamples of implementations of the present invention rather than toprovide an exhaustive list of all possible implementations of thepresent invention.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared and otherwisemanipulated. It has proven convenient at times, principally for reasonsof common usage, to refer to these signals as bits, values, elements,symbols, characters, terms, numbers or the like. The blocks describedherein may be hardware, software, firmware, or a combination thereof.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “detecting,” “initiating,” “determining,” “continuing,”“halting,” “receiving,” “recording,” or the like, refer to the actionsand processes of a computing system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(e.g., electronic) quantities within the computing system's registersand memories into other data similarly represented as physicalquantities within the computing system memories or registers or othersuch information storage, transmission or media devices.

The words “example” or “exemplary” are used herein to mean serving as anexample, instance or illustration. Any aspect or design described hereinas “example’ or “exemplary” is not necessarily to be construed aspreferred or advantageous over other aspects or designs. Rather, use ofthe words “example” or “exemplary” is intended to present concepts in aconcrete fashion. As used in this application, the term “or” is intendedto mean an inclusive “or” rather than an exclusive “or.” That is, unlessspecified otherwise, or clear from context, “X includes A or B” isintended to mean any of the natural inclusive permutations. That is, ifX includes A; X includes B; or X includes both A and B, then “X includesA or B” is satisfied under any of the foregoing instances. In addition,the articles “a” and “an” as used in this application and the appendedclaims should generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform. Moreover, use of the term “an implementation” or “oneimplementation” or “an implementation” or “one implementation”throughout is not intended to mean the same implementation orimplementation unless described as such. Also, the terms “first,”“second,” “third,” “fourth,” etc. as used herein are meant as labels todistinguish among different elements and may not necessarily have anordinal meaning according to their numerical designation.

Implementations described herein may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise a computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a non-transitorycomputer-readable storage medium, such as, but not limited to, any typeof disk including floppy disks, optical disks, CD-ROMs andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flashmemory, or any type of media suitable for storing electronicinstructions. The term “computer-readable storage medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database and/or associated caches and servers) that storethe one or more sets of instructions. The term “computer-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that causes the machine to perform any one or more of themethodologies of the present implementations. The term“computer-readable storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, optical media,magnetic media, any medium that is capable of storing a set ofinstructions for execution by the machine and that causes the machine toperform any one or more of the methodologies of the presentimplementations.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various computing systemsmay be used with programs in accordance with the teachings herein, or itmay prove convenient to construct a more specialized apparatus toperform the operations. The required structure for a variety of thesesystems will appear from the description below. In addition, the presentimplementations are not described with reference to any particularprogramming language. It will be appreciated that a variety ofprogramming languages may be used to implement the teachings of theimplementations as described herein.

The above description sets forth numerous specific details such asexamples of specific systems, components, methods and so forth, in orderto provide a good understanding of several implementations. It will beapparent to one skilled in the art, however, that at least someimplementations may be practiced without these specific details. Inother instances, well-known components or methods are not described indetail or are presented in simple block diagram format in order to avoidunnecessarily obscuring the present implementations. Thus, the specificdetails set forth above are merely exemplary. Particular implementationsmay vary from these exemplary details and still be contemplated to bewithin the scope of the present implementations.

It is to be understood that the above description is intended to beillustrative and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the present implementations should,therefore, be determined with reference to the appended claims, alongwith the full scope of equivalents to which such claims are entitled.

The disclosure above encompasses multiple distinct embodiments withindependent utility. While these embodiments have been disclosed in aparticular form, the specific embodiments disclosed and illustratedabove are not to be considered in a limiting sense as numerousvariations are possible. The subject matter of the embodiments includesthe novel and non-obvious combinations and sub-combinations of thevarious elements, features, functions and/or properties disclosed aboveand inherent to those skilled in the art pertaining to such embodiments.Where the disclosure or subsequently filed claims recite “a” element, “afirst” element, or any such equivalent term, the disclosure or claims isto be understood to incorporate one or more such elements, neitherrequiring nor excluding two or more such elements.

Applicant(s) reserves the right to submit claims directed tocombinations and sub-combinations of the disclosed embodiments that arebelieved to be novel and non-obvious. Embodiments embodied in othercombinations and sub-combinations of features, functions, elementsand/or properties may be claimed through amendment of those claims orpresentation of new claims in the present application or in a relatedapplication. Such amended or new claims, whether they are directed tothe same embodiment or a different embodiment and whether they aredifferent, broader, narrower or equal in scope to the original claims,are to be considered within the subject matter of the embodimentsdescribed herein.

1. A device, comprising: a wearable display configured to: attach to ahead of a user; and display an augmented reality environment to theuser, wherein the augmented reality environment includes a virtualobject; a sensor configured to: detect a position of a first digit of afirst hand of the user; and detect a position of a second digit of thefirst hand of the user; a processing device coupled to the wearabledisplay and the sensor, wherein the processing device is configured to:identify a first fingertip of the first digit; identify a secondfingertip of the second digit; determine that the first fingertip andthe second fingertip are in an open pinch position at a first point intime; determine that the first fingertip and the second fingertip are ina closed pinch position at a second point in time, wherein the secondpoint in time is after the first point in time; determine that the firstfingertip and the second fingertip are in the closed pinch position fora first threshold period of time; and in response to that the firstfingertip and the second fingertip are in the closed pinch position forthe first threshold period of time, execute a tap instruction associatedwith the closed pinch position, wherein the tap instruction correspondswith a tap instruction executed on a touch sensor of another device. 2.The device of claim 1, wherein the processing device is furtherconfigured to: determine a midpoint between the first fingertip and thesecond fingertip at the first point in time; and display a cursor at themidpoint between the first fingertip and the second fingertip.
 3. Thedevice of claim 2, wherein to identify the first fingertip and thesecond fingertip, the processing device is further configured to:capture, by the sensor, an image or a video of the first hand; anddetermine geometric information of the first hand in the image or aframe of the video using a hand skeleton detection algorithm to identifyhand skeleton key points.
 4. The device of claim 3, wherein theprocessing device is further configured to identify the positions of thefirst digit, the second digit, the first fingertip, or the secondfingertip based on the hand skeleton key points.
 5. The device of claim1, wherein: the sensor is further configured to: detect a position of athird digit of a second hand of the user; detect a position of a fourthdigit of the second hand of the user; the processing device coupled tothe wearable display and the sensor, wherein the processing device isconfigured to: identify a third fingertip of the third digit; identify afourth fingertip of the fourth digit; determine that the third fingertipand the fourth fingertip are in the open pinch position at the firstpoint in time; determine that the third fingertip and the thirdfingertip are in the closed pinch position at the second point in time;determine that the first fingertip and the second fingertip are in theclosed pinch position; and in response to that the first fingertip, thesecond fingertip, the third fingertip, and the fourth fingertip being inthe closed pinch position, execute another instruction.
 6. The device ofclaim 5, wherein the other instruction is a zoom instruction to enlargeor shrink a size of the virtual object.
 7. The device of claim 6,wherein the processing device is configured to: determine that the firstfingertip and the second fingertip are moving away from the thirdfingertip and the fourth fingertip; and execute the zoom instruction toenlarge the virtual object.
 8. The device of claim 6, wherein theprocessing device is configured to: determine that the first fingertipand the second fingertip are moving towards the third fingertip and thefourth fingertip; and execute the zoom instruction to shrink the virtualobject.
 9. The device of claim 5, wherein: the other instruction is arotate instruction; and the processing device is configured to:determine that the first fingertip, the second fingertip, the thirdfingertip, and the fourth fingertip rotate about an axis; and rotate thevirtual object about the axis relative to the rotation of the firstfingertip, the second fingertip, the third fingertip, and the fourthfingertip rotate about the axis.
 10. The device of claim 1, wherein thewearable display, the sensor, and the processing device are integratedinto smart glasses.
 11. The device of claim 1, wherein the processingdevice is further configured to: determine that the first fingertip andthe second fingertip are in the closed pinch position for a secondthreshold period of time; and in response to that the first fingertipand the second fingertip are in the closed pinch position for the secondthreshold period of time, execute another instruction associated withthe closed pinch position, wherein the other instruction is differentthan the tap instruction.
 12. A method, comprising: detecting, by asensor, a position of a first digit of a hand of a user; detecting, bythe sensor, a position of a second digit of the hand of the user;identifying, by a processing device, a first fingertip of the firstdigit; identifying, by the processing device, a second fingertip of thesecond digit; determining, by the processing device, that the firstfingertip and the second fingertip are in an open pinch position at afirst point in time; determining, by the processing device, that thefirst fingertip and the second fingertip are in a closed pinch positionat a second point in time, wherein the second point in time is after thefirst point in time; determining, by the processing device, that thefirst fingertip and the second fingertip are in the closed pinchposition for a threshold period of time; and in response to that thefirst fingertip and the second fingertip are in the closed pinchposition for the threshold period of time, executing a tap instructionassociated with the closed pinch position, wherein the tap instructioncorresponds with a tap instruction executed on a touch sensor of anotherdevice.
 13. The method of claim 12, further comprising: determining, bythe processing device, a midpoint between the first fingertip and thesecond fingertip at the first point in time; and displaying, by awearable display, a cursor at the midpoint between the first fingertipand the second fingertip.
 14. The method of claim 13, whereinidentifying the first fingertip and the second fingertip comprising:capturing, by the sensor, an image or a video of the hand; anddetermining geometric information of the hand in the image or a frame ofthe video using a hand skeleton detection algorithm to identify handskeleton key points.
 15. The method of claim 14, further comprisingidentifying the positions of the first digit, the second digit, thefirst fingertip, or the second fingertip based on the hand skeleton keypoints.
 16. A device, comprising: a wearable display configured to:attach to a head of a user; and display an augmented reality environmentto the user, wherein the augmented reality environment includes avirtual object; a sensor configured to: detect a position of a firstdigit of a hand of the user; and detect a position of a second digit ofthe hand of the user; a processing device coupled to the wearabledisplay and the sensor, wherein the processing device is configured to:identify a first fingertip of the first digit; identify a secondfingertip of the second digit; determine that the first fingertip andthe second fingertip are in an open pinch position at a first point intime; and display, by the wearable display, a cursor at a midpointbetween the first fingertip and the second fingertip.
 17. The device ofclaim 16, wherein the processing device is configured to: determine thatthe first fingertip and the second fingertip are in a closed pinchposition at a second point in time; determine that the first fingertipand the second fingertip are in the closed pinch position for a firstthreshold period of time; and in response to that the first fingertipand the second fingertip are in the closed pinch position for the firstthreshold period of time, execute a tap instruction associated with theclosed pinch position, wherein the tap instruction corresponds with atap instruction executed on a touch sensor of another device.
 18. Thedevice of claim 17, wherein the processing device is configured to:identify the virtual object corresponding to a location of the cursor;and select the virtual object to execute the tap instruction onto inresponse to that the first fingertip and the second fingertip are in theclosed pinch position for a threshold period of time.
 19. The device ofclaim 17, wherein the tap instruction is an instruction associated witha single tap on the touch sensor or a double tap on the touch sensor.20. The device of claim 17, wherein the processing device is configuredto: determine that the first fingertip and the second fingertip are in aclosed pinch position for a second threshold period of time; and inresponse to that the first fingertip and the second fingertip are in theclosed pinch position for the second threshold period of time, executeanother instruction associated with the closed pinch position, whereinthe other instruction is different than the tap instruction.